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ABSTRACT 



This paper discusses impact studies, an emerging evaluation 
and planning strategy evolving from required USDA reporting for extension 
services and agricultural experiment stations to a framework for 
evaluation/assessment planning. Impact studies can be useful for evaluating 
unit and individual performance, resource allocation, and planning. In higher 
education, impact studies can be used to demonstrate how work done by people 
on campus is worth continuing and how the work merits support from federal 
and state governments, funding agencies, students, parents, and the general 
population. The paper begins by defining impact studies, explaining that they 
provide a distinction between outcomes (achievements) and impacts. The next 
section discusses the development of a system of gathering and using impacts, 
focusing on problems and procedures and highlighting seven principles (e.g., 
if there is a lot of support for this activity, find something else to do; 
start with a likely success; become a master scavenger; commit to long-term 
follow-up; and expect expenses) . The paper presents examples of impact 
evaluation, discusses relationships to other effectiveness and evaluation 
models, and notes pilot project results. (Contains 14 references.) (SM) 
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IMPACT ANALYSES: Concepts and Methods 

Abstract 

This paper discusses an emerging evaluation and planning strategy evolving 
from required USD A reporting for extension services and agricultural 
experiment stations to a framework for evaluation/assessment/planning. 
"Impact" studies derive from definitions developed by Claude Bennett of the 
USDA and are useful for evaluation of unit and individual performance, 
resource allocation, and planning. The paper develops definitions, presents 
examples, discusses relationship to other effectiveness and evaluation 
models, and discusses pilot project results. 
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IMPACT ANALYSES: Concepts and Methods 



CONCEPTS AND MATTERS OF DEFINITION 
The nineteen eighties and nineties were a time of shifting to the age of 
public scrutiny and accountability from Madaus, Scriven, and Stufflebeam's 
age of professionalism (1983). Accountability standards began to change 
from questions which could be answered by readily obtainable data about 
workload, outcomes, and accomplishments to questions which demand 
information useful for deciding institutional direction, allocating resources, 
and especially for communicating to external agencies. In a sense, the 
question is being altered from one of how we can demonstrate that people on 
our campuses work to one of how can we show that the work they do is 
worth continuing and that it merits the support of state and federal 
governments, funding agencies public and private, students and parents, and 
a general population which understands little of the broad functions of 
higher education. 

What follows is an attempt to articulate a way of thinking and 
communicating about what makes our work important enough to warrant 
continued support. Looking at issues related to valuing research, public 
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service, and instruction from the view of their impacts permits one to move 
away from imposed and pre-defined standards and formulas as well as away 
from asking our critics and our constituents to take our self judgements on 
faith. The rather Cartesian "1 think, therefore 1 am worthy of support" does 
not command much attention in a "What have you done for me, lately" 
world 

For years, state level extension services and agricultural experiment stations 
have wrestled with federal requirements to document the "impacts" of their 
efforts ( Bennett, 1976, 1979). An impact study responds to these questions: 
after all the work has been done, what were the results and were they 
important, or "so what", and what population is affected, or "who cares" 
(Ernst et al, 1998). Impacts can be intended or accidental, beneficial or 
destructive. The major foci in the extension literature on impact studies are 
societal (often meaning physical or mental well-being), economic, and 
environmental conditions (Bennett, Bennett, 1976; Bennett and Rockwell, 

1 995). For purposes of the university at large, knowledge systems (feedback 
into research and graduate training systems) and education should be added. 
Impact thinking is grounded in the belief that what we spend our time on 
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must be truly important; it asks the question of how different the world 
would be if this activity hadn't happened (Brandenburg, 1 998). 

Bennett's thinking defined the nature of impact projects. Elements of his 
hierarchy of activities involved in creating change are (bottom to top) 
resources; activities; participation; reactions; changes in knowledge, 
attitudes, skills, and aspirations (KASA); changes in practices, and the 
resulting changes in the environmental, social, or economic conditions. He 
points out that as one goes up his scale the cost of evaluation goes up and 
that at that top level, it goes up a lot. Expense and effort go a long way 
toward explaining why that level of evaluation is rare. 

As extension offices and experiment stations developed systems of 
compliance, impact discussions spread to the research/development and 
instructional communities within related agricultural colleges. (Oregon 
State University's "Oregon Invests!" (Dutson and Evans, 1 998) was to my 
knowledge the first large scale project developed to meet these needs. Web 
sites from the Universities of Nebraska and Minnesota are also interesting 
examples.) The terminology and concerns of impact thinking are beginning 
to affect discussions at the university level. There seems to be some 
recognition that shifting standards of accountability cannot be met by 
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traditional measures of effort and accomplishment. For example, the 
assessment movement of the past twenty years has represented an attempt at 
moving past asking the question of how much did we do in the classroom to 
asking if what we did made some worthy difference. Traditional productivity 
measures have become topics for discussion. For example, "books 
published" is a valued traditional measure of productivity, in part because it 
is relatively easy to count, but many institutions have given more weight to 
such categorical variables than makes sense. Clearly, there is a difference 
between Rachel Carson's Silent Spring and a book I co-edited some years 
ago on data issues. One has had incredible impact and continues to generate 
"impacts" even after fifty years. The other is, at best, little known. Thinking 
in these terms makes it difficult to justify counting each as "one" in the 
books published column. 



A key definitional issue, then, is the distinction between "outcomes" (or 
"achievements") and "impacts". In the 1980's "outcomes" became a catchall 
name for attempts to assess results, but the term, as it has been used, 
encompasses too much. For purposes of this discussion of an impact 
approach, it is productive to think in terms of just two domains: precursors 
and impacts. Outcomes, in this way of looking at matters, are part of the 
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domain of precursors, which includes all the efforts and all the products 
(papers, pamphlets, books, speeches, pesticide-sprayers, concerts, new 
varieties of grass seed, grant dollars, etc.), preparation, training, knowledge 
gained, attitudes changed, and the efforts it took to produce them (Bennett's 
hierarchy helps to make this clear). This doesn't mean that precursors are not 
important; it just says that they are not impacts and that they do not address 
some critical accountability issues. In academia our tradition has been to 
justify our work on the basis of such precursor/causal elements. Our 
approach has been to focus on internal audiences to show that we did the 
work of the academy (SCH, papers, books) and that the work had merit 
according to the standard of collegial approval (grants, prestigious 
publications, awards). The demand from the outside (funding agencies, 
taxpayers, legislators), though, drives the need to respond to the question of 
"What have you done for me lately" and to do so in plain language. 
Traditionally, the academy has asked its publics to take on faith the merits of 
its work. In the current atmosphere, honest and well expressed appraisal 
within the framework of a clearly articulated value system will be a key to 
survival. 
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Impact analysis is related to forty years of thinking about evaluation 
(Chronbach, 1963, Dressel, 1976, Madaus, Scriven, and Stufflebeam, 1983), 
to cost benefit analysis, institutional effectiveness, logic models, academic 
assessment and evaluation, planning models, and so on. In general, impact 
analysis is the step past which most approaches stop (that their authors did 
not intend for them to stop short of impacts is a separate issue. Dressel 
(1976) clearly articulates the need to address long term implications. For 
reasons discussed below, reviews often stop short of such goals). Evaluating 
from the perspective of impacts (Any students of Bennett's reading this far 
will note that 1 have diverged from his work. Looking at the complexities of 
university efforts and operations versus looking at the focused and planned 
activities of extension operations is the basis for this difference) differs from 
most other approaches by asking what happened, not what did we want to 
happen, nor what did we plan to happen, nor what do we think should have 
happened, nor what happened that made us look good, nor how much effort 
did we expend. Those questions can be asked after we know what really 
happened. Finding out what really happened is not easy and has drawbacks: 
it is expensive, risky, and often requires long term study. Its merits are that it 
can incorporate qualitative and quantitative information, it is a powerful 
management tool, and it is a powerful communications tool, unlike many 
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traditional measures. Impact analysis' ability to show what did not work as 
well as what did will be seen as a drawback or a merit depending on one's 
situation, and there will be resistance because it can be used for personnel 
and program allocation decisions (Dressel, 1976). Raising critical questions 
and provoking their answers, both of which happen as one reviews impact 
analyses, cannot please everyone, but they can provide meaningful guidance 
for reshaping our institutions. In addition, they provide material useful for 
presenting results appropriate to the project rather than reports shaped by 
standardized formats. 

DEVELOPING A SYSTEM: PROBLEMS, PROCEDURES, 
PRINCIPLES 

There are two real problems in trying to develop a system of gathering and 
using impacts. The first is that most of the people one deals with have no 
concepts around which to build a discussion. Most have been part of a 
system which values outputs for so long that it is hard to shake loose from 
the belief that articles or books or credit hours represent the only ways of 
looking at productivity. 

The other side of this coin is that others do have concepts, ones which are 
rigid and narrow. Many believe that an "impact system" is a website or 
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other communications process for displaying "success stories" for purposes 
of public relations. Although the results of an impact project may be used in 
that way, they may be used in many other ways, and, if honestly done, not 
all will lend themselves to public relations. 

1 find myself thinking of impacts in terms of recipes for salmon. The first 
step in any recipe for salmon, albeit rarely stated, is this: first you must get a 
salmon. Catch it, buy it, borrow, or steal it; no salmon, no point in talking 
about what you should do with it. Now, if you actually have a salmon, you 
can prepare it in many ways, plain and fancy, raw or cooked, grilled or 
poached, and so on. You may even choose to toss it in the garbage and go 
out for steak or tofu, as you see fit. 

Impacts are much the same. First you must determine what they are. Then 
you may determine how each will be used. The majority will have a place in 
program review. Some will have a place in communicating to funding agents 
and other constituents ways in which programs have met their needs or 
solved their problems. Most will have a place in evaluation. None will be 
of value to anyone until they have been "caught". Getting past all the 
discussion of what to do with the material long before there is any is a 
challenge. 
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A thorough, competent, and honest approach will certainly reveal some 
project which has had a negative impact (perhaps an economically positive 
move has resulted in a serious pollution problem, for example). To suggest 
that this should go automatically to the front page of an impact web site or to 
the front page of the Inquirer is nai ve. To suggest that this should be the 
start of a program to fix the problem is, of course, appropriate and desirable. 

Pilot project experience suggests the following: 

PRINCIPLE 1 . If there is not a lot of support for this activity, find 
something else to do. It's difficult enough to make major cultural changes if 
there is support from the top (Moses discovered this to be true), but without 
help at the department chair levels and consultation with faculty, you are just 
another person asking others to do more work. 

PRINCIPLE 2: Start with a likely success. The impacts of technology 
transfer are relatively easy to determine. They tend to be concrete, estimable 
in the early stages, and measurable during follow up. The impacts of 
knowledge transfer are much more difficult, although Bennett's hierarchy 
provides real guidance. Therefore, start with technology transfer. When you 
get to the issue of classroom instruction, aside from the vast body of 
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literature on assessment, Scheffer and Rubenfeld's work on critical thinking 
in nursing education is not limited to their discipline. 

PRINCIPLE 3: Become a master scavenger. Much of what you want exists; 
the problem is finding it. Ask no one to duplicate reporting efforts. Use 
existing material to every degree possible. Asking faculty to modify an 
existing report, web page, required information, or other existing 
information may not gel you a really bad reaction. On the other hand, 
asking for the same information which someone has already filled out (often 
more than once) will cause people to dig in their heels, and that will be the 
end of that. 

PRINCIPLE 4: Analysis of impacts should be an important part of 
planning. One of the issues of our lime will be (or is, depending on your 
situation) that of a coordinated and targeted program at one extreme versus a 
policy of chasing funding wherever it lakes a department at the other. 

Asking the impact question can provide a standard of measure for discussing 
this issue. It will not solve the issue, just provide some measures, but that's a 
move forward (see principle six) . Adding impact concerns to annual 
reviews and promotion and tenure reviews within departments can make 
them a regular part of the planning cycle at little additional effort cost on the 
part of department chairs. 
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PRINCIPLE 5: Commit to long term follow-up. "Impact" and follow-up are 
virtually synonymous. For good or bad, only the perspective of hind sight 
permits one to view impacts realistically. Not doing the long term follow-up 
robs policy makers at the next go around of information they need. Consider 
the history of grants to increase milk production, surplus dairy products, 
subsidies, and long term health issues related to animal fats in the diet, for 
example. 

Curriculum reform would be a good case for an impact study. Baseline 

t 

measures followed by long term study of those who were affected by the 
reform would make some sense. For the most part, however, curriculum 
reform decisions are based on the "Think Method" (Willson, 1957) 

PRINCIPLE 6: The pilot project suggests that what you are likely to get 

requires that you think in terms of three phases: 

Phase 1 . In phase 1 you have to ask people to think about evaluating what 
they've already done. Part of the job in this phase is to get people thinking 
in terms of building an impact based design into the beginning of their next 
projects or programs. 

This phase produces a mixed bag of responses, many of which will not go 
past the point at which we tend to wash our hands of projects or which will 
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veer off into "accomplishments". A good example is that of a project to 
train low income families to become active in community politics. The 
project coordinator reported that the "impact" was that all who stayed in the 
training program were now trained to become active. Whether any of them 
became active or was effective was not considered. Whether the community 
changed was not addressed. 

Some reports will come in as very complete impact reports. They will tend 
to be ones for which there are clear economic measures. 

Phase 2. This is tough, because one has to go back to faculty and 
administrators and tell them that they did not do well enough. In some 
cases, they didn't do well enough because not enough time had passed for 
effects to occur. Helping them to articulate potential impacts and commit to 
actually looking after an appropriate time is critical. It can be a hard sell 
until people begin to sec that their impact statements are useful to them and 
are being used to improve their situations. 

Phase 3. Reliance on self reporting is essential in any large organization. 
Pragmatic issues of cost, time, and significance make this so. However, 
many projects/programs will require more thorough and more credible 
evaluation. Finding competent evaluators is not a problem. Rewarding 
them sufficiently that they can afford to participate is. Having a team of 
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evaluators which can assist, counsel, and provide external perspective is 
necessary to get past problems ranging from inexperience to a "no bad 
news" orientation. 

PRINCIPLE 7. Determining institutional values is essential to making 
use of impact results. When you have discovered that there is little if any 
measurable impact to the university's carillon, that alumni contributions 
seem unrelated to its being played or not played, that it makes just as good a 
logo whether the bells ring or not, the decision to fund the carrilloneur and 
annual maintenance must come down to institutional values. The same 
application of institutional values is true for the decision to support basic 
research projects versus applied, courses in Urdu versus extra freshman 
writing labs, and so on. Determining the impact provides information for 
discussion about such decisions. Institutional values determine how that 
information is viewed. 

PRINCIPLE 8. This is going to cost. 



SOME PHASE 1 EXAMPLES. These are abstracts of reports on several 

recent or current projects with editorial comment. 

1 . Air curtain sprayer development. Pesticide reduction in orchard 
industries. 

Reduction of pesticide sprayed on orchard crops through 
innovative sprayer development has economic and 
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environmental impacts estimated at up to $22 million . New 
industrial development and better health in orchard industry 
communities as well as increased fruit production and 
decreased expense for pesticides are among the anticipated 
benefits. 

This is an interesting project which has the potential for 
environmental, social, and economic impacts, but the orchard 
industries in the areas of the country which need it the most are 
in a period of economic difficulty and may not be able to afford 
the technology. Impacts are dependent on the adoption of the 
technology; until that happens estimates and potentials are all 
you can talk about. 

1 Michigan Groundwater Stewardship Program Evaluation 

As a result of evaluations, the risk of pesticide and nitrogen 
contamination in the groundwater in Michigan is being reduced. 
Seventy-five percent of the 8,600 farmers who have completed 
on farm risk assessments have implemented at least one 
stewardship practice. 4,1 00 abandoned wells have been 
plugged, eliminating direct conduits for contaminants to reach 
groundwater; 4,000 working wells have been protected through 
the addition of anti-backflow devices; 3,600 leak eliminating 
improvements have been made to chemical application 
equipment; 276,000 pounds of unused pesticides have been 
collected for safe disposal; and 302,000 plastic pesticide 
containers have been recycled saving over 3,1 00 yards of 
landfill space. Follow up studies will determine long term 
impact on the environment. 

Recognition of the need for follow up is important. Each of 
these actions seems very positive. The question of whether 
they made a difference remains. 

1 . Hispanic/farm and farmworker community study 

Preliminary study identifying the problems and issues 
confronting the Hispanic population of the County. Meetings 
will be held to address the results of the study. Follow-up will 
be necessary to assess impact. 
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Once again, the recognition that the project is a part of 
something larger and that it does not represent a stopping point 
is important. In other contexts it would be logged as an activity 
or accomplishment. Recognizing that it is not at an end point 
should be part of a planning process which asks how resources 
will be used. 
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